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(57) Abstract: A system is provided for seam- 
less synchronization through asynchronous in- 
tegration of Internet content with broadcast con- 
tent within a portable handheld device (14), such 
as a remote control or a personal digital assistant 
(PDA). The handheld device (14) receives pro- 
gram-related data from the television (12) while 
the user is watching a program. The program- 
related data places the handheld device (14) in 
sync with the program the user is watching and 
augments the user's enjoyment of the program 
by providing supplemental broadcast informa- 
tion or additional interactive features. The hand- 
held device (14) is also configured to retrieve ad- 
ditional supplemental information from at least 
one content source distinct from the broadcast 
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ASYNCHRONOUS INTEGRATION OF PORTABLE 
HANDHELD DEVICE 

FIELD OF THE INVENTION 
[0001] The present invention relates generally to interactive 
television, and nndre particularly, to a system for seamlessly synchronizing 
Internet content with broadcast content within a portable handheld device. 

BACKGROUND OF THE INVENTION 

[0002] There is considerable Interest today in ways to Integrate 
supplemental content information with broadcast media. For example, 
broadcasters seek ways to engage viewers by providing interactive content from a 
disparate content source, such as the Internet, that coincides with or supplements 
the program being broadcast. However, cunrent interaction with the Internet is 
quite limited today, because the typical TV viewer does not have a personal 
computer conveniently located in the TV viewing room, nor is the typical TV 
viewer able to navigate to the appropriate Internet addresses while watching the 
TV program. In addition, current interaction approaches try to synchronize the 
supplemental internet content with the broadcast program content b^ controlling 
the timing of tlie respective information sources. Tliis approacli has ,a number of 
problems, among them being that synchronization cannot occur if the broadcast 
content is delayed as through recording or timeshifting. 

[0003] The present invention addresses these concerns by 
seamlessly synchronizing the supplemental Internet content with the broadcast 
content within a portable handheld device, such as a remote control or a personal 
digital assistance (PDA). The handheld device receives program-related data 
from the television while the user is watching a program. The program-related 
data places the handheld device in sync with the program the user is watching 
and augments the user's enjoyment of the program by providing supplemental 
broadcast Information or additional interactive features. The handheld device is 
also configured to retrieve additional supplemental broadcast information from at 
least one content source distinct from the broadcast source. 
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[0004] For a more complete understanding of the invention, its 
objects and advantages refer to the following specification and to the 
accompanying drawings. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

[0005] Figure 1 is a diagram depicting an exemplary interactive TV 
system in accordance with the present invention; 

[0006] Figure 2 illustrates exemplary advertisement information 
which may be displayed to the viewer of a broadcast commercial via the handheld 
10 device in accordance with the present invention; 

[0007] Figure 3 is diagram illustrating how additional supplemental 
broadcast data is retrieved from a disparate content source in accordance with the 
present invention; 

[0008] Figure 4 illustrates an exemplary electronic coupon which 
1 5 may be downloaded to the handheld device of the present invention; and 

[0009] Figure 5 is a block diagram of an exemplary architecture used 
to implement the interactive TV system of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
20 [0010] Figure 1 illustrates an exemplary interactive TV system 10 

that achieves synchronization through asynchronous integration of information 
from disparate sources. The interactive TV system 10 is generally comprised of a 
television 12, a portable handheld device 14, and at least one disparate media 
content source 16, such as the Internet. The television 12 is configured to receive 
25 audio/visual content from a broadcast source 13 and deliver the audio/visual 
content to a viewer as is well known in the art. The audio/visual content received 
from the broadcast source 13 may also be encoded with supplemental broadcast 
data, including but not limited to electronic program guide data, closed caption 
data, program-related data, etc. It is readily understood that at least some of the 
30 supplemental broadcast data may be delivered to the viewer via the television 12. 

[0011] To enhance the viewing experience, at least some of the 
supplemental broadcast data may be transmitted via a wireless communication 
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link to the handheld device 14. In accx)rdance with the present invention, the 
television 12 (or accompanying set-top box) may be operable to transmit 
supplemental broadcast data that correlates to the broadcast program currently 
being viewed. The supplemental broadcast data may then serve as the basis for 
5 providing interactive features to the viewer. 

[0012] For example, advertisement information corresponding to a 
broadcast commercial may be displayed to the viewer by the handheld device 14 
as shown in Figure 2. In this example, the viewer is watching a broadcast 
commercial from Pizza Pal while watching an ongoing television program 

10 corresponding to a basketball game (as shown in the subtitle under the "Live" 
menu). Additional product description information and a related word game may 
simultaneously appear on the user interface of the handheld device 14. It is 
envisioned that other related advertisement information, such as product name, 
product manufacture, local retail locations including address and phone number, 

15 web site address, etc., may also be provided to the viewer via the handheld 
device 14. 

[0013] In another example, information related to the broadcast 
program the user Is watching may appear on the handheld device 14. Program- 
related information may Include anecdotal information about the program, 

20 biographical information for the program's actors, program producer's 
commentary, questions from a quiz show, statistics associated with a broadcasted 
sporting event, etc. It is readily understood that other types of Interactive features 
may be implemented on the handheld device 14, 

[0014] To further enhance the viewers interactive experience, the 

25 handheld device 14 is configured to retrieve additional supplemental broadcast 
data from a disparate content source 16 as shown in Figure 3. To do so, the 
supplemental broadcast data 32 received from the broadcast source 13 is 
embedded with trigger data. Upon receipt of the supplemental broadcast data 32, 
the handheld device 14 Is operable to parse out any trigger data embedded 

30 therein and store it in a data store residing in the handheld device 14. The trigger 
data is then used to retrieve additional supplemental Information from the 
disparate content source 16. 
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[0015] A request for addftional information 34 is fomnulated by the 
handheld device 14. The request 34 Is transmitted via a wireless communication 
link to the disparate content source 16. In response to the request, additional 
supplemental information 36 is sent from the disparate content source 16 to the 
5 handheld device 14. It is important to note that the request may be sent 
asynchronously from the receipt of the supplemental broadcast data from the 
broadcast source and that the additional supplemental infonnatlon may be sent 
asynchronously from the request for the information. The additional supplemental 
information is then used by the handheld device 14 to further enhance the viewers 
10 interactive experience. 

[0016] An enhanced interactive experience is further described 
below. During a broadcast television commercial, an electronic coupon 
corresponding to the advertised goods or services may be available to the viewer. 
Supplemental broadcast data transmitted to the handheld device may initiate a 
1 5 display that alerts the viewer as to the availability of the coupon. In one instance, 
the viewer then requests retrieval of the coupon using the handheld device 14. 
Altematively. the availability of the coupon to the viewer may be based on a game 
related to the commercial, such as the word game shown in Figure 2. In this 
instance, successful completion of the game by the viewer prompts retrieval of the 
20 coupon. In either instance, the electronic coupon is subsequently downloaded to 
the handheld device 14 from a disparate content source 16 as described above. 
An exemplary coupon is shown in Figure 4. It is envisioned that the electronic 
coupon may be redeemable at either an on-line shopping venue or a local retail 
establishment. 

25 [0017] An exemplary system architecture for implementing the 

present invention is described in relation to Figure 5. The primary components of 
the system architecture 50 may include a television 52. a personal digital 
attendant (PDA) 54. and a home gateway 56. While the following description is 
provided with reference to a PDA, it is readily understood that the broader aspects 

30- of the present invention are applicable to remote control devices, cellular phones 
and other portable handheld devices. 
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[0018] In a conventional analog domain, the vertical blanking interval 
(VBI) is the portion of the television signal that is used to carry supplemental 
broadcast data. In digital broadcasting, supplemental broadcast data can be 
transmitted within the digital transport stream. The system architecture 50 may 
5 further include a set-top box 58 for decoding supplementary data from the TV 
signal (only supplementary data mechanism is drawn). Although a set-top box is 
presently preferred, it is readily understood that the decoding function may be 
incorporated Into the TV or into another add-on device (e.g., DVD recorder) which 
may be used in conjunction with the television. It is also understood that other 

10 data encoding techniques are also within the scope of the present invention. 

[0019] The set-top box 58 is adapted to receive the incoming TV 
signal from the broadcast source. In order to retrieve supplementary data, the 
set-top box 58 includes a data decoder 64. The data decoder 64 is a hardware 
component that pulls supplementary data off of the incoming TV signal. In analog 

15 broadcast domain, the data decoder 64 can be a VBI decoder. In digital domain, 
the data decoder 64 can be a section filter implemented within a transport stream 
decoder. The data decoder 64 may be configured to read a specified channel and 
decode the data transmitted in the specified channel. 

[0020] Since supplementary data is transmitted at only certain time 

20 intervals on each channel, it is not instantaneously available upon request from 
the TV signal. Rather, the system Is designed to buffer supplementary data as it 
becomes available. The data decoder 64 can be configured to constantly retrieve 
all available data from the TV signal and stores it in a data store 68 residing on the 
set-top box. 

25 [0021] The set-top box 58 further includes a filter module 70 and a 

wireless transceiver device 72. The filter module 70 is a software module that 
retrieves supplementary data from the data buffer 68 that correlates to the 
ongoing activities on the handheld that involve supplementary information from 
the broadcast source. For example, when an interactive game is transmitted to the 

30 handheld in association with a TV commercial, upon the user playing the game, 
the electronic coupon, which is stored in data buffer 68, is downloaded to the 
handheld. The user may also pre-configure the data filter module 70 to send 
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supplementary Information only In certain categories. For advertisement, however, 
the user may not be able to block all commercials, but can pre-select a minimum 
number of favorite categories of commercials such as tools, gardens, and/or 
health products. 

5 [0022] The supplementary broadcast data Is then transmitted via the 

wireless transceiver device 72 to the PDA 14. The data is preferably transmitted 
by the wireless transceiver device 72 using the Bluetooth protocol, IEEE 802.11b 
protocol or some other known wireless communication protocol. The same 
principle can equivalently be applied in both analog and digital TV broadcasting, in 
10 which the data encoding and decoding are different. 

[0023] The PDA 54 serves as the focal point for the viewer's 
interactive television experience. To enhance the viewing experience, at least 
one application 80 residing on the PDA 14 is configured to continuously stream 
supplementary broadcast data which con-elates to the audio/visual content being 
15 viewed on the television. Applications are preferably developed using Java or 
some other known application development tool. 

[0024] In one embodiment, supplementary data is received at the 
PDA 14 using a wireless transceiver device 74 and then stored in a 
supplementary database 76. To ease interaction with other applications residing 
20 on the PDA, supplementary data is preferably stored in XML forniat. A data 
parser 78 serves to retrieve the supplementary data from the supplementary 
database 76 and translate it into a format useable by the application 80. Various 
commercially available XML-based parsers may be used to implement the data 
parser. 

25 [0025] In order to retrieve additional supplemental data from the 

Internet, the data parser 78 detects any trigger data which may be embedded in 
the supplementary broadcast data. Upon detection, the data parser 78 parses out 
the trigger data and passes it to a supplemental request module 82. The 
supplemental request module 82 then fomnulates the request for any additional 

30 supplemental information. 

[0026] In one embodiment, the additional supplemental infomiatlon 
is retrieved from the Internet via a home gateway 56. In particular, the 
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supplemental Information is retrieved by a web browser 84 using retrieval 
techniques well known in the art. The supplemental Infonnatlon may be passed in 
real-time by the web browser 84 to the application 80, where it is displayed to the 
viewer. Alternatively, the supplemental infonmatlon may be stored In a data store 

5 residing on the PDA for subsequent presentation to the viewer. In either case, the 
supplemental Infonnation may be embedded with data that may be used to 
synchronize the presentation of the supplemental infonnation with the audio/visual 
content being viewed on the television. 

[00271 The home gateway 18 may include a wireless transceiver 92, 

10 a router 94, and a cable modem 96 as is well known in the art. However, it is 
readily understood that other means (e.g., a cellular or satellite communication 
link) may be employed to access the Internet. Alternatively, the additional 
supplemental infomriation may be retrieved from a local content source (e.g., SD 
or CF memory cards) and/or other remote content sources. 

1 5 [0028] One skilled in the art will readily recognize that some or all of 

the functionality embedded In the PDA may be perfomied in one or more other 
devices associated with the broadcast environment, including but not limited to a 
digital video recorder, the television, the settop box. the home gateway or other 
known devices. Likewise, it is readily understood that at least some of the 

20 functionality embedded in the settop box or the television may be integrated Into 
the PDA. 

[0029] The description of the Invention is merely exemplary in nature 
and, thus, variations that do not depart from the gist of the invention are intended 
to be within the scope of the invention. Such variations are not to be regarded as 
25 a departure from the spirit and scope of the invention. 
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CLAIMS 

What is claimed is: 

1. A method for asynchronously accessing supplementary media 
content based on broadcast media content received from a broadcast signal for 
use witti a handheld device, comprising: 

receiving trigger infonmation extracted from the broadcast media 

content; 

obtaining supplementary media content from a disparate source of 

media content; and 

accessing the supplementary media content based on the trigger 

information. 

2. The method of claim 1, comprising receiving broadcast media 
content having trigger information embedded therein. 

3. The method of claim 1, comprising extracting the trigger information 
from the broadcast media content. 

4. The method of claim 1, comprising delivering broadcast media 
content received from the broadcast signal. 

5. The method of claim 1, comprising communicating trigger 
infomnation extracted from the broadcast signal to the handheld device. 

6. The method of claim 1, comprising delivering the supplementary 
media content to a consumer of the broadcast media content. 

7. The method of claim 6, comprising employing a communication 
capability of the hand-held device to deliver the supplementary media content to 
the consumer of the broadcast media content. 

8. The method of claim 1, comprising obtaining the supplementary 
media content associated with broadcast media content prior to delivery of the 
broadcast media content to a consumer. 

9. The method of claim 1, comprising storing the supplementary media 
content In a memory of the handheld device. 

10. The method of claim 1, comprising obtaining the supplementary 
media content from a disparate source of media content based on the trigger 
information. 
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11. The method of claim 10, wherein the disparate source corresponds 
to at least one of the Intemet, an external data source, and a mobile network. 

12. The method of claim 1, wherein said receiving trigger Information 
includes receiving an identification tag adapted to identify the broadcast media 
content. 

13. The method of claim 1, wherein said receiving trigger Infonnation 
includes receiving an identification tag adapted to identify the supplementary 
media content. 

14. The method of claim 1 , wherein said obtaining supplementary media 
content includes obtaining an identification tag. 

1 5. The method of claim 1 , wherein said obtaining supplementary media 
content Includes obtaining descriptive text. 

16. The method of claim 1, wherein said obtaining supplementary media 
content includes obtaining binary Infomnatlon. 

17. The method of claim 16, wherein said obtaining binary infomiation 
includes obtaining an image file. 

18. The method of claim 16. wherein said obtaining binary infomnatlon 

includes obtaining a media file. 

19. The method of claim 16, wherein said obtaining binary Infomnatlon 
includes obtaining software operable to execute an interactive game. 

20. The method of claim 1 , wherein said obtaining supplementary media 
content includes obtaining hyper text markup language. 

21. The method of claim 1, wherein said wherein said obtaining 
supplementary media content includes obtaining a link to a location of 
supplementary media content on a communications network. 

22. The method of claim 1 , wherein said obtaining supplementary media 
content includes obtaining compressed infomnatlon. 

23. The method of claim 1 , wherein said obtaining supplementary media 
content includes obtaining uncompressed infomnatlon. 

24. The method of claim 1, comprising retrieving supplementary media 
content stored in memory of the handheld device based on ongoing activities of 
the handheld device relating to the supplementary media content. 
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25. The method of claim 1, comprising retrieving supplementary media 
content stored in memory of the handheld device based on user-defined 
categories relating to desired supplementary media content. 

26. A supplementary media content access system employing a 
handheld device, comprising: 

an Input of the handheld device receiving trigger infomiation 
extracted from the broadcast media content; 

a supplemental request module obtaining supplementary media 
content from a disparate source of media content; and 

a parser accessing the supplementary media content based on the 

trigger Informatipn. 

27. The system of claim 26, comprising a set top box receiving 
broadcast media content having trigger information embedded therein. 

28. The system of claim 26, comprising a decoder extracting the trigger 
information from the broadcast media content. 

29. The system of claim 26, comprising a media delivery device 
delivering broadcast media content received from the broadcast signal. 

30. The system of claim 26, comprising a set top box communicating 
trigger information extracted from the broadcast signal to the handheld device. 

31. The system of claim 26, comprising a media delivery device 
delivering the supplementary media content to a consumer of the broadcast media 
content. 

32. The system of claim 31 , comprising a graphic user interface of the 
handheld device delivering the supplementary media content to the consumer of 
the broadcast media content. 

33. The system of claim 26, wherein said supplemental request module 
is adapted to obtain the supplementary media content associated with broadcast 
media content prior to delivery of the broadcast media content to a consumer. 

34. The system of claim 26, comprising a memory of the handheld 
device storing the supplementary media content. 
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35. The system of claim 26, wherein said supplemental request module 
is adapted to obtain the supplementary media content from a disparate source of 
media content based on the trigger information. 

36. The method of claim 35, wherein the disparate source con-esponds 
to at least one of the Internet, an external data source, and a mobile network. 

37. The system of claim 26, wherein said input receives an identification 
tag adapted to identify the broadcast media content. 

38. The system of claim 26, wherein said input receives an identification 
tag adapted to identify the supplementary media content. 

39. The system of claim 26, wherein said supplemental request module 
obtains an identification tag. 

40. The system of claim 26, wherein said supplemental request module 

obtains descriptive text. 

41. The system of claim 26, wherein said supplemental request module 

obtains binary information. 

42. The system of claim 41, wherein said supplemental request module 

obtains an image file. 

43. The system of claim 41, wherein said supplemental request module 

obtains a media file. 

44. The system of claim 41 , wherein said supplemental request module 
obtains software operable to execute an interactive game. 

45. The system of claim 26, wherein said supplemental request module 
obtains hyper text marl<up language. 

46. The system of claim 26. wherein said wherein said supplemental 
request module obtains a link to a location of supplementary media content on a 
communications network. 

47. The system of claim 26, wherein said supplemental request module 

obtains compressed information. 

48. The system of claim 26, wherein said supplemental request module 
obtains uncompressed information. 

49. The system of claim 26, comprising a filter module retrieving 
supplementary media content stored in memory of the handheld device based on 
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ongoing activities of the liandiield device relating to tiie supplementary media 
content. 

50. The system of claim 26, comprising a filter module retrieving 
supplementary media content stored in memory of the handheld device based on 

5 user-defined categories relating to desired supplementary media content. 

51 . A method of asynchronously providing information to a portable 
device from a broadcast source and an information source, comprising: 

supplying first Information to said portable device from said 
broadcast source; 

10 requesting second Information from said information source based 

on said first information; and 

supplying said second information to said portable device. 

52. A method of asynchronously using information in a portable device, 
comprising: 

1 5 using said portable device to receive first information from a 

broadcast source; 

using said portable device to provide control instructions to a second 
system based on said first information; 

using said second system to perform an operation based on said 
20 control instructions, 

53. A method of asynchronously using information from a broadcast 
source, comprising: 

using a tuner to selectively input first information from a broadcast 
source into a portable device; 
25 using said portable device to provide control instructions to a second 

system based on said first Information; 

using said second system to perform an operation based on said 
control instructions. 
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AMENDED CLAIMS 
[received by the International Bureau on 22 September 2003 (22.09.03); 
original claim 1 amended, claims 2-53 unchanged ] 

What is claimed is: 

1. A method for asynchronously accessing supplementary media 
content based on broadcast media content received 1ix>m a broadcast signal for use 

5 with a handheld device, comprising: 

receiving trigger information extracted from the broadcast media 

content; 

obtaining supplementary media content from a disparate soun:e of 
media content; and 

10 accessing the supplementary media content based on the trigger 

information, wherein said accessing occurs asyncronously, without simultaneous 
connection to a source of the broadcast signal and the disparate source of media 
content during said accessing. 

2. The method of claim 1 , comprising receiving broadcast media content 
1 5 having trigger information embedded therein. 

3. The method of claim 1, comprising extracting the trigger information 
from the broadcast media content 

4. The method of daim 1, oompri^ng delivering broadcast media 
content received firom the broadcast signal. 

20 5. The method of daim 1, comprising communicailng t^er information 

extracted from the broadcast signal to tiie handheld device. 

6. The method of claim 1. comprising delivering the supplementary 
media content to a consumer of the broadcast media content. 

7. The method of claim 6. comprising employing a communication 
25 capability of the hand-held device to deliver the supplementary media content to 

the consumer of Vne broadcast media content. 

8. The method of daim 1, comprising obtaining the supplementary 
media content associated whh broadcast media content prior to defivary of the 
broadcast media content to a consumer. 

30 9. The method of claim 1, comprising storing the supplementary media 

content in a memory of the handheld device. 

10. The method of daim 1, comprising obtaining the supplementary 
media content from a disparate source of media content baaed on the trigger 
information. 

AMENDED SHEET (ARTICLE 19) 
13 
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